This page is to discuss the next version of the log and event tracing system. The collection system is fairly robust and works well with a number of different scenarios already (syslog, windows event logs, log4j files) and can easily be written to handle network listener based events or perhaps code enhancement based eventing. Where we are currently lacking is in the long term persistent storage and analysis features.

Storage

For storage, first we need to get out of the database and into a distributed store to disk model with compression and indexing. This screams for a solution like HDFS. Others have already solved this problem using HDFS and there even exists the Chukwa project that is designed to implement a self contained solution. Chukwa is interesting because it is a solution already in use and has tackled many of the necessary topics. One downside may be the difficulty in integrating with the RHQ infrastructure to gain the most value from the cohesive management strategy tying other types of data together. E.g. the rhq agent already manages itself, upgrades itself, has reliable, secure delivery, and collects log and events data in a way that attaches it to our deep inventory system. 

Analysis and Reporting

We want to be able to perform both large scale searches of the data as well as correlated analysis of specific message rates or transformation to quantitative data sets. Externalizing the data set would remove it one step from our inventory system which may make some of the analysis more difficult. (e.g. Search the logs for this "cluster" which is an RHQ inventory concept) It also would need considerable attention on the alerting integration for best results.

Requirements

• Total audit store (all log messages stored and available) (i.e. scale to TBs of data, 1000's of monitored servers)

• Reliable, highly-available persistence (preferably selectable CAP scenarios)

• Distributed with geographic sensitivity so we don't have to ship full log contents between datacenters

• Seamless installation, upgrade and management

• Users views of reconstituted log files for a resource, cluster or arbitrary group

• Search log files for specific messages, summaries, groupings, time-periods, etc.

• Continue to integrate into alerting (and expand the fanciness)

• Secondary analysis

  • Count of types of messages graphed over time, over arbitrary sets of resources

  • Processing messages for included information (e.g. output timings, appache response codes, etc)

• Do all of the above (or as much as possible) with real-time performance